class: center, middle, inverse, title-slide # A workflow for open and reproducible MRI studies ## Discussion series on “Human Research Data in Practice” ### Lennart Wittkuhn (<a href="mailto:wittkuhn@mpib-berlin.mpg.de" class="email">wittkuhn@mpib-berlin.mpg.de</a>) ### Max Planck Research Group NeuroCode<br>Max Planck Institute for Human Development, Berlin, Germany ### 22th of June 2021 --- # About #### About me - PhD student at the [Max Planck Research Group "NeuroCode"](https://www.mpib-berlin.mpg.de/research/research-groups/mprg-neurocode) at the [Max Planck Institute for Human Development](https://www.mpib-berlin.mpg.de/en) in Berlin - Research: I study the role of fast neural reactivation ("replay") in decision-making in humans using fMRI - Member of the MPIB's working group on research data management -- #### About this presentation - Rendered presentation publicly available via - Source code publicly available on GitHub - This presentation contains links to external resources. I do not take responsibility for the accuracy, legality or content of the external site or for that of subsequent links. - I am happy for any feedback or suggestion via [email](mailto:wittkuhn@mpib-berlin.mpg.de) or [GitHub issues](https://github.com/lnnrtwttkhn/talk-rdm/issues). Thank you! 🙏 --- # The philosophy .pull-left[ *"An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures."* Buckheit & Donoho (1995), paraphrasing Jon Claerbout ] .pull-right[ <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#https://reproducibility.org/blog/wp-content/uploads/2015/08/claerbout1.jpg" alt="Jon Claerbout &lt;br&gt;Geophysicist at Stanford University" width="50%" /> <p class="caption">Jon Claerbout <br>Geophysicist at Stanford University</p> </div> ] ??? - Jon Claerbout, a distinguished exploration geophysicist at Stanford University - He has also pointed out that we have reached a point where solutions are available - it is now possible to publish computational research that is really reproducible by others. --- # The challenge ??? - Data is produced through code (e.g., task code) - Data is manipulated by code and new data is generated - Mapping between input and output data - This happens using specific software in specific versions --- # "Practice" of research code and data management - "*Where is the data?*" -- - "*Can I see your code?*" -- - "*Which version of the code and data did you use to produce this result?*" -- - "*What is the difference between `data_version1_edit.csv` and `data_version8_new_final.csv`?*" -- - "*I get different results on my machine ...*" -- - "*But it worked when I ran it last month?!*" --- # The solution - Code, data and computational environments change all the time! - &rarr; We need **version-control** --- # Our paper -- #### Two-sentence summary: > Non-invasive measurement of fast neural activity with spatial precision in humans is difficult. Here, the authors show how fMRI can be used to detect sub-second neural sequences in a localized fashion and report fast replay of images in visual cortex that occurred independently of the hippocampus. --- # Data management using DataLad: Overview #### From Wittkuhn & Schuck, 2021, *Nature Communications* (see [Data Availability statement](https://www.nature.com/articles/s41467-021-21970-2#data-availability)): > *"We publicly share all data used in this study. Data and code management was realized using DataLad [version 0.13.0142, for details, see https://www.datalad.org/].*" -- - Publicly available project website: https://wittkuhn.mpib.berlin/highspeed/ - All individual datasets can be found at: https://gin.g-node.org/lnnrtwttkhn - Each dataset is associated with a unique URL and a Digital Object Identifier (DOI). -- #### All data? -- - MRI and behavioral data adhering to the [BIDS standard](https://bids.neuroimaging.io/) ([GitHub](https://github.com/lnnrtwttkhn/highspeed-bids), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-bids), [DOI](https://doi.org/10.12751/g-node.4ivuv8)) - MRI quality metrics and reports based on [MRIQC](https://mriqc.readthedocs.io/en/stable/) ([GitHub](https://github.com/lnnrtwttkhn/highspeed-mriqc), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-mriqc), [DOI](https://doi.org/10.12751/g-node.0vmyuh)) - preprocessed MRI data using [fMRIPrep](https://fmriprep.org/en/stable/), ([GitHub](https://github.com/lnnrtwttkhn/highspeed-fmriprep), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-fmriprep), [DOI](https://doi.org/10.12751/g-node.0ft06t)) - binarized anatomical masks used for feature selection ([GitHub](https://github.com/lnnrtwttkhn/highspeed-masks), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-masks), [DOI](https://doi.org/10.12751/g-node.omirok)) - first-level GLM results used for feature selection ([GitHub](https://github.com/lnnrtwttkhn/highspeed-glm), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-glm), [DOI](https://doi.org/10.12751/g-node.d21zpv)) - results of the multivariate decoding approach ([GitHub](https://github.com/lnnrtwttkhn/highspeed-decoding), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-decoding), [DOI](https://doi.org/10.12751/g-node.9zft1r)) - unprocessed data of the behavioral task acquired during MRI acquisition ([GitHub](https://github.com/lnnrtwttkhn/highspeed-data-behavior), [GIN](https://gin.g-node.org/lnnrtwttkhn/highspeed-data-behavior), [DOI](https://doi.org/10.12751/g-node.p7dabb)) \> 1.5 TB in total --- # Data sharing via GIN <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#https://gin.g-node.org/img/favicon.png" alt="https://gin.g-node.org/" width="10%" /> <p class="caption">https://gin.g-node.org/</p> </div> > "*GIN is [...] a web-accessible repository store of your data based on git and git-annex that you can access securely anywhere you desire while keeping your data in sync, backed up and easily accessible [...]"* #### Advantages of GIN (non-exhaustive list) - free and open-source - supports private and public repositories - publicly funded by the Federal Ministry of Education and Research (BMBF) - servers are on German land (near Munich, Germany) - provides Digital Object Identifiers (DOIs) (details [here](https://gin.g-node.org/G-Node/Info/wiki/DOI)) - allows to set own license (details [here](https://gin.g-node.org/G-Node/Info/wiki/Licensing)) - DataLad plays perfectly with GIN (details [here](https://handbook.datalad.org/en/latest/basics/101-139-gin.html)) --- # Publishing a DataLad dataset to GIN in only 4 steps Step 1: Create a dataset ```bash datalad create my_dataset ``` -- Step 2: Save data into the dataset ```bash datalad save -m "add data to dataset" ``` -- Step 3: Add the GIN remote ("sibling") ```bash datalad siblings add -d . --name gin --url git@gin.g-node.org:/my_username/my_dataset.git ``` -- Step 4: Transfer the dataset to GIN ```bash datalad push --to gin ``` -- Done! 🎉 --- # Multiple remotes <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/publishing_network_publishparts2.svg" alt="http://handbook.datalad.org/en/latest/basics/101-138-sharethirdparty.html" width="60%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/101-138-sharethirdparty.html</p> </div> --- # Code sharing using Git and DataLad --- # What is DataLad? #### What is DataLad (see [10,000 feet overview](http://handbook.datalad.org/en/latest/intro/executive_summary.html))? - Free, open-open source **command-line tool** - Building on top of **Git** and **git-annex**, DataLad allows you to **version control arbitrarily large files** in datasets. - DataLad doesn't "lock you in" -- - I'm "only" an enthusiastic user of DataLad! 😊 --- # DataLad: What is a dataset? > *"A dataset is a directory on a computer that DataLad manages."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/dataset.svg" alt="http://handbook.datalad.org/en/latest/basics/101-101-create.html" width="40%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/101-101-create.html</p> </div> > "*You can create new, empty datasets [..] and populate them, or transform existing directories into datasets.*" --- # DataLad: Version-control arbitrarily large files > *"Building on top of Git and git-annex, DataLad allows you to version control arbitrarily large files in datasets."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/local_wf.svg" alt="http://handbook.datalad.org/en/latest/basics/101-107-summary.html" width="40%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/101-107-summary.html</p> </div> > *"[...] keep track of revisions of data of any size, and view, interact with or restore any version of your dataset [...]."* --- # DataLad: Dataset consumption and collaboration > *"DataLad lets you consume datasets provided by others, and collaborate with them."* > *"You can install existing datasets and update them from their sources, or create sibling datasets that you can publish updates to and pull updates from for collaboration and data sharing."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/collaboration.svg" alt="http://handbook.datalad.org/en/latest/basics/101-120-summary.html" width="60%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/101-120-summary.html</p> </div> --- # DatLad: Dataset linkage > *"Datasets can contain other datasets (subdatasets), nested arbitrarily deep."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/linkage_subds1.svg" alt="http://handbook.datalad.org/en/latest/basics/101-106-nesting.html" width="70%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/101-106-nesting.html</p> </div> > *"Each dataset has an independent [...] history, but can be registered at a precise version in higher-level datasets. This allows to combine datasets and to perform commands recursively across a hierarchy of datasets, and it is the basis for advanced provenance capture abilities."* --- # DataLad: Full provenance capture and reproducibility > *"DataLad allows to **capture full provenance**: The origin of datasets, the origin of files obtained from web sources, complete machine-readable and automatically reproducible records of how files were created (including software environments)."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/reproducible_execution.svg" alt="http://handbook.datalad.org/en/latest/basics/101-113-summary.html" width="50%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/101-113-summary.html</p> </div> > *"You or your collaborators can thus re-obtain or reproducibly **recompute content with a single command**, and make use of extensive provenance of dataset content **(who created it, when, and how?)**."* --- # DataLad: Third party service integration > *"**Export datasets to third party services** such as GitHub, GitLab, or Figshare with built-in commands."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/thirdparty.svg" alt="http://handbook.datalad.org/en/latest/basics/basics-thirdparty.html" width="50%" /> <p class="caption">http://handbook.datalad.org/en/latest/basics/basics-thirdparty.html</p> </div> > *"Alternatively, you can use a multitude of other available third party services such as Dropbox, Google Drive, Amazon S3, owncloud, or many more that DataLad datasets are compatible with."* --- # DataLad: Metadata handling > *"**Extract, aggregate, and query dataset metadata.** This allows to automatically obtain metadata according to different metadata standards (EXIF, XMP, ID3, BIDS, DICOM, NIfTI1, ...), store this metadata in a portable format, share it, and search dataset contents."* <div class="figure" style="text-align: center"> <img src="data:image/png;base64,#http://handbook.datalad.org/en/latest/_images/metadata_prov_imaging.svg" alt="http://docs.datalad.org/en/stable/metadata.html" width="90%" /> <p class="caption">http://docs.datalad.org/en/stable/metadata.html</p> </div> --- # Resources on DataLad - The [DataLad Handbook](http://handbook.datalad.org/en/latest/) is an incredibly extensive resource - YouTube video: ["What is DataLad"](https://www.youtube.com/watch?v=IN0vowZ67vs) - YouTube video: Michael Hanke: ["How to introduce data management technology without sinking the ship?"](https://www.youtube.com/watch?v=uH75kYgwLH4) --- # Outlook, challenges and discussions > *"He [Jon Claerbout] has also pointed out that we have reached a point where solutions are available - it is now possible to publish computational research that is really reproducible by others.*" Buckheit & Donoho (1995), describing how Jon Claerbout and his team shared CD-ROMs with interactive code that could regenerate the figures in their books - in the *early 90s* *(addition in parantheses)* -- #### All technical solutions are available! The long-term challenges are: - moving towards a "culture of reproducibility" (cf. Russ Poldrack, see e.g., [here](https://www.youtube.com/watch?v=XjW3t-qXAiE)) - changing incentives / funding schemes - education, education, education - "slow science" --- # Challenge: Relationship between code and data *"Which code produced which data?"* -- #### Solutions - [datalad run](http://docs.datalad.org/en/stable/generated/man/datalad-run.html) > `datalad run` *"[...] will record a shell command, and save all changes this command triggered in the dataset – be that new files or changes to existing files."* (see [details](http://handbook.datalad.org/en/latest/basics/basics-run.html) in the DataLad handbook) - [GNU Make](https://www.gnu.org/software/make/) > *"Make enables [...] to build and install your package without knowing the details of how that is done -- because these details are recorded in the makefile that you supply."* > *"Make figures out automatically which files it needs to update, based on which source files have changed. It also automatically determines the proper order for updating files [...]"* --- # Challenge: Implementing a Data User Agreement (DUA) #### From Wittkuhn & Schuck, 2021, project website (see section on [license information](https://wittkuhn.mpib.berlin/highspeed/#license-information)): > If you download any of the published data, please complete our Data User Agreeement (DUA). The Data User Agreement (DUA) we use for this study, was taken from the Open Brain Consent project, distributed under Creative Commons Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0). - based on templates and recommendations of the [Open Brain Consent](https://open-brain-consent.readthedocs.io/en/stable/) project (licensed [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en)) - optional for data from Wittkuhn & Schuck, 2021 - not possible to implement mandatory DUA on GIN --- # Challenge: --- # Thank you!